[Chapter 23] 23.2 Parts of a Queued Message
Author: Bryan Costales & Eric Allman
Date: 1999-01-06T19:31:48Z

#

| Chapter 23 The Queue |

## 23.2 Parts of a Queued Message

When a message is stored in the queue, it is split into pieces. Each of those pieces is stored as a separate file in the queue directory. That is, the header and other information about the message are stored in one file, while the body (the data) is stored in another. All told, six different types of files may appear in the queue directory. The type of each is denoted by the first two letters of the filenames. Each filename begins with a single letter followed by an `f` character. The complete list is shown in Table 23.1.

Table 23.1: Queue File Types
File |  | Description
--- | --- | ---
`df` | Section 23.2.2, "The Data (Message Body) File: df" | Data (message body)
`lf` | Section 23.2.3, "The Lock File (obsolete as of V5.62): lf" | Lock file (obsolete as of V5.62)
`nf` | Section 23.2.4, "The ID Creation File (obsolete as of V5.62): nf" | ID creation file (obsolete as of V5.62)
`tf` | Section 23.2.6, "The Temporary qf Rewrite Image: tf" | Temporary qf rewrite image
`xf` | Section 23.2.7, "The Transcript File: xf" | Transcript file
`qf` | Section 23.2.5, "The Queue Control File: qf" | Queue control file (and headers)

The complete form for each filename is

>     Xfident

The `X` is one of the leading letters shown in Table 23.1. The `f` is the constant letter `f`. The `ident` is a unique queue identifier associated with each mail message.

In the following sections we first describe the identifier that is common to all the queue file parts, then describe each file type in alphabetical order. The internal details of the `qf` file can vary depending on the version of sendmail, so it is discussed separately at the end of this chapter.

### 23.2.1 The Queue Identifier

To ensure that new filenames are not the same as the names of files that may already be in the queue, sendmail uses the following pattern for each new `ident`:

>     AApid prior to V8.6
>     hourAApid beginning with V8.6

Here, `pid` is the process identification number of the incarnation of sendmail that is trying to create the file. Because sendmail often fork(2)'s to process the queue, that `pid` is likely to be unique, resulting in a unique `ident`. For V8 sendmail an extra letter prefixes the `AA`. Shown as `hour`, it is an uppercase letter that corresponds to the hour (in a 24-hour clock) that the identifier was created. For example, a file created in hour three of the day will have a `D` prefixed (the hour begins at midnight with `A`). [2]

> [2] Programs should not depend on the lead letter actually encoding the hour. It is intended only to ensure that all identifiers be unique within any 24-hour period and as an aid to scripts that need to extract information from log files.

If sendmail cannot create an exclusive filename (because a file with that identifier already exists), it clocks the second `A` of the `AA` to a `B` and tries again. It continues this process, clocking the right-hand letter from `A` to `Z` and the left-hand letter from `A` to `~` until it succeeds:

>     AA             start
>     AB             second try
>     AC             third try
>      ... and so on
>     ~W
>     ~X
>     ~Y             last try
>     ~Z             failure

If it never succeeds, the `ident` ultimately looks like the following and sendmail has failed:

>     hour~Zpid

This `ident` is unlikely to ever appear, because the clocking provides for over 1600 possibilities.

All the files associated with a given mail message share the same `ident` as a part of their filenames. The individual files associated with a single mail message differ only in the first letter of their names.

### 23.2.2 The Data (Message Body) File: df

All mail messages are composed of a header and a body. When queued, the body is stored in the df file.

Traditionally, the message body could contain only characters that had the high (most significant) bit turned off (cleared, set to 0). But under V8 sendmail, with a version 2 or higher configuration file (see Section 27.5, "The V Configuration Command"), the high bit is left as is until delivery (whereupon the `F=7` delivery-agent flag, see Section 30.8.4, F=7, determines whether or not that bit will be stripped during delivery).

Because the message body can contain sensitive or personal information, the `df` file should be protected from reading by ordinary users. If the queue directory is world readable, then the `TempFileMode` (`F`) option (see Section 34.8.68, TempFileMode (F)) should specify minimum permissions (such as 0600) for queued files. But if the queue directory is protected by both narrow permissions and a secure machine, the `TempFileMode` (`F`) option may be relaxed for easier administration.

There is currently no plan to provide for encryption of `df` files. If you are concerned about the privacy of your message, you should use an end-to-end encryption package (not discussed in this book).

### 23.2.3 The Lock File (obsolete as of V5.62): lf

When old versions of sendmail process a queued message (attempt to redeliver it) they create an empty lock file. That lock file is necessary to signal to other running sendmail processes that the mail message is busy so that they won't try to deliver the message too. Current versions simply flock(2) or fcntl(2) the `qf` file.

#### 23.2.3.1 Current style file locking

The method that sendmail uses to initially create an exclusive lock when first queueing a file is twofold. First it attempts to creat(2) the file with the argument

>     O_CREAT|O_WRONLY|O_EXCL

If that succeeds, it then attempts to lock the file. If HASFLOCK (see Section 18.8.9, HAS...) is defined when sendmail is compiled, the file is locked with flock(2). Otherwise, it is locked with a fcntl(2) `F_SETLK` argument.

#### 23.2.3.2 Locks shown when printing the queue

When mailq is run (or the `-bp` command-line switch is given to sendmail), the contents of the queue are listed. In that listing, an asterisk that appears to the right of an identifier indicates that a lock exists on the message:

>                     Mail Queue (1 request)
>     -Q-ID- -Size- --Q-Time--- ------Sender/Recipient------
>     MAA17445*   126 Fri Apr 17 10:17 <gw@wash.dc.gov>
>                                     <ben@franklin.edu>
>            note

#### 23.2.3.3 Locks can get stuck

Occasionally, a file will become locked and remain that way for a long time. One indication of a stuck lock is a series of syslog messages about a given identifier:

>     Apr 12 00:33:38 ourhost sendmail[641]: AA00614: locked
>     Apr 12 01:22:14 ourhost sendmail[976]: AA00614: locked
>     Apr 12 02:49:23 ourhost sendmail[3251]: AA00976: locked
>     Apr 12 02:49:51 ourhost sendmail[5977]: AA00614: locked
>     Apr 12 03:53:05 ourhost sendmail[9839]: AA00614: locked

An occasional lock message, such as `AA00976` in the third line above, is normal. But when an identifier is continually reporting as locked (like the `AA00614` lines), an orphaned lock may exist and should be investigated. Use ps(1) to look for lines that list queue file identifiers:

>     root      5338 160  -AA00614 To wash.dc.gov (sendmail)

This shows that the queued mail message, whose identifier is `AA00614`, is currently being processed. It the lock on that file is stuck, consider killing the sendmail that is processing it.

### 23.2.4 The ID Creation File (obsolete as of V5.62): nf

Old versions of sendmail used an `nf` file when creating a message identifier to avoid race conditions. [3] But contemporary versions of sendmail create the queue identifier when first creating the `qf` file. The `nf` file is obsolete.

> [3] Historical footnote: This stems from the days when the only atomic file-system call was link(2).

### 23.2.5 The Queue Control File: qf

A queued mail message is composed of two primary parts. The `df` file contains the message body. The `qf` file contains the message header.

In addition to the header, the `qf` file also contains all the information necessary to:

- Deliver the message. It contains the sender's address and a list of recipient addresses.

- Order message processing. It contains a priority that determines the current message's position in a queue run of many messages.

- Expire the message. It contains the date that the message was originally queued. That date is used to time out a message.

- Explain the message. It contains the reason that the message is in the queue and possibly the error that caused it to be queued.

The `qf` file is line-oriented, with one item of information per line. Each line begins with a single uppercase character (the code letter), which specifies the contents of the line. Each code letter is then followed by the information appropriate to the letter. The code letters and their meanings are shown in Table 23.2 of Section 23.9, "The qf File Internals".

Here is an example of a version 1 (for V8.8 sendmail) `qf` file:

>     V1
>     T826845694
>     K0
>     N0
>     P30016
>     I7/4/20
>     $_you@localhost
>     MDeferred: Host wash.dc.gov is down
>     Syou@your.domain
>     RPFD:george@wash.dc.gov
>     RPFD:jefferson
>     H?P?Return-Path: you@your.domain
>     HReceived: (from you@your.domain) by your.domain (8.8.4/8.8.4)
>             id QAA06571 for george@wash.dc.gov; Thu, 14 Mar 1996 16:21:34 -0700 (MST)
>     H?D?Date: Thu, 14 Mar 1996 16:21:34 -0700 (MST)
>     H?F?From: Your Name <you@your.domain>
>     H?x?Full-Name: Your Name
>     H?M?Message-Id: <199603142321.QAA06571@your.domain>
>     HSubject: foo
>     .

This fictional `qf` file shows the information that will be used to send a mail message from `you@your.domain` (the `S` line) to two recipients: `george@wash.dc.gov` and `jefferson` (the `R` lines). It also shows the various headers that appear in that message (the `H` lines). We discuss the individual lines of the `qf` file in at the end of this chapter.

### 23.2.6 The Temporary qf Rewrite Image: tf

When processing a queued message, it is often necessary for sendmail to modify the contents of the `qf` file. This usually occurs if delivery has failed or if delivery for only a part of the recipient list succeeded. In either event, at least the message priority needs to be incremented.

To prevent damage to the original `qf` file, sendmail makes changes to a temporary copy of that file. The temporary copy has the same queue identifier as the original, but its name begins with a `t`.

After the `tf` file has been successfully written and closed, sendmail calls rename(2) to replace the original with the copy. If the renaming fails, sendmail syslog(3)'s at LOG_CRIT a message like the following:

>     cannot rename(tfAA00000, qfAA00000), df=dfAA00000

Failure to rename is an unusual, but serious, problem: A queued message has been processed, but its `qf` file contains old and incorrect information. This failure may, for example, indicate a hardware error, a corrupted queue directory, or that the system administrator accidentally removed the queue directory.

### 23.2.7 The Transcript File: xf

A given mail message may be destined for many recipients, requiring different delivery agents. During the process of delivery, error messages (such as "User unknown" and "Permission denied") can be printed back to sendmail by each delivery agent.

While calling the necessary delivery agents, sendmail saves all the error messages it receives in a temporary file. The name of that temporary file begins with the letters `xf`. After all delivery agents have been called, sendmail returns any collected error messages to the sender and deletes the temporary `xf` file. If there are no errors, the empty `xf` file is silently deleted. The `-d51.104` debugging switch (see Section 37.5.171, -d51.104) can be used to prevent deletion of the `xf` file.

23.1 Overview of the Queue |  | 23.3 A Bogus qf File (V8 only): Qf